[BE]: Update 12.6 builds to CUDA 12.6.3 #141433

Skylion007 · 2024-11-23T17:00:22Z

Update CUDA 12.6 to Update 3 and make cusparse-lt 0.6.3? #141365 Was going to leave some comments on #141365, but though it was just faster to open a PR here.

related to #138440

pytorch-bot · 2024-11-23T17:00:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141433

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 17 Unrelated Failures

As of commit b61ff06 with merge base b75bb64 ():

NEW FAILURES - The following jobs have failed:

Build manywheel docker images for s390x / build-docker-cpu-s390x (gh)
no space left on device
linux-binary-manywheel / manywheel-py3_10-rocm6_2-test (gh)
+ exit 1
linux-binary-manywheel / manywheel-py3_11-rocm6_1-test (gh)
##[error]Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_9-rocm6_2-test (gh)
+ exit 1
macos-arm64-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-build (gh)
cp: /Users/runner/work/pytorch/pytorch/pytorch/any_wheel/torch/lib/libomp.dylib: No such file or directory
windows-binary-wheel / wheel-py3_12-cpu-test (gh)
'pip' is not recognized as an internal or external command,

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_1-shared-with-deps-pre-cxx11-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_2-shared-with-deps-pre-cxx11-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_10-rocm6_1-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_10-xpu-test (gh) (similar failure)
ERROR: Could not find a version that satisfies the requirement pytorch-triton-xpu==3.2.0+git91b14bf5; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 3.0.0+1b2f15840e, 3.0.0+cc981feba1, 3.1.0+91b14bf559)
linux-binary-manywheel / manywheel-py3_11-rocm6_2-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_11-xpu-test (gh) (similar failure)
ERROR: Could not find a version that satisfies the requirement pytorch-triton-xpu==3.2.0+git91b14bf5; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 3.0.0+1b2f15840e, 3.0.0+cc981feba1, 3.1.0+91b14bf559)
linux-binary-manywheel / manywheel-py3_12-rocm6_1-test (gh) (similar failure)
++ sed -ne 's/.*D_GLIBCXX_USE_CXX11_ABI=\(.\)".*/\1/p'
linux-binary-manywheel / manywheel-py3_12-rocm6_2-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_12-xpu-test (gh) (similar failure)
ERROR: Could not find a version that satisfies the requirement pytorch-triton-xpu==3.2.0+git91b14bf5; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 3.0.0+1b2f15840e, 3.0.0+cc981feba1, 3.1.0+91b14bf559)
linux-binary-manywheel / manywheel-py3_13-xpu-test (gh) (similar failure)
/opt/intel/oneapi/compiler/latest/env/vars.sh: line 263: OCL_ICD_FILENAMES: unbound variable
linux-binary-manywheel / manywheel-py3_9-rocm6_1-test (gh) (similar failure)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_9-xpu-test (gh) (similar failure)
ERROR: Could not find a version that satisfies the requirement pytorch-triton-xpu==3.2.0+git91b14bf5; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 3.0.0+1b2f15840e, 3.0.0+cc981feba1, 3.1.0+91b14bf559)
windows-binary-wheel / wheel-py3_10-xpu-test (gh) (similar failure)
OSError: [WinError 126] The specified module could not be found. Error loading "C:\actions-runner\_work\pytorch\pytorch\builder\windows\Python\lib\site-packages\torch\lib\c10_xpu.dll" or one of its dependencies.
windows-binary-wheel / wheel-py3_11-xpu-test (gh) (similar failure)
OSError: [WinError 126] The specified module could not be found. Error loading "C:\actions-runner\_work\pytorch\pytorch\builder\windows\Python\Lib\site-packages\torch\lib\c10_xpu.dll" or one of its dependencies.
windows-binary-wheel / wheel-py3_12-xpu-test (gh) (similar failure)
OSError: [WinError 126] The specified module could not be found. Error loading "C:\actions-runner\_work\pytorch\pytorch\builder\windows\Python\Lib\site-packages\torch\lib\c10_xpu.dll" or one of its dependencies.
windows-binary-wheel / wheel-py3_13-xpu-test (gh) (similar failure)
OSError: [WinError 126] The specified module could not be found. Error loading "C:\actions-runner\_work\pytorch\pytorch\builder\windows\Python\Lib\site-packages\torch\lib\c10_xpu.dll" or one of its dependencies.
windows-binary-wheel / wheel-py3_9-xpu-test (gh) (similar failure)
OSError: [WinError 126] The specified module could not be found. Error loading "C:\actions-runner\_work\pytorch\pytorch\builder\windows\Python\lib\site-packages\torch\lib\c10_xpu.dll" or one of its dependencies.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Skylion007 · 2024-11-23T17:13:05Z

.github/scripts/generate_binary_build_matrix.py

@atalman I'll probably need this uploaded to the cu126 s3 bucket. Changes are from the https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-toolkit-major-component-versions list

atalman

lgtm. Lets wait for green signal to land

tinglvv

Regarding the pypi matrix packages, should be nvidia-nvjitlink-cu12==12.6.85 and nvidia-cublas-cu12==12.6.4.1 for 12.6.3

tinglvv · 2024-11-27T18:42:17Z

Adding the ciflow/binaries label to test the x86 nightly wheel.

Skylion007 · 2024-11-27T21:46:58Z

@atalman Looks like we'll need those binaries uploaded to S3

atalman · 2024-11-27T22:27:48Z

@Skylion007 let me upload this now

Skylion007 · 2024-11-28T15:29:29Z

Remaining failures appear unrelated.

Skylion007 · 2024-11-28T16:38:46Z

@pytorchbot merge -i

pytorchmergebot · 2024-11-28T16:41:41Z

Merge started

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-11-28T16:42:36Z

Merge failed

Reason: 17 jobs have failed, first few of them are: windows-binary-wheel / wheel-py3_9-xpu-test, windows-binary-wheel / wheel-py3_11-xpu-test, windows-binary-wheel / wheel-py3_13-xpu-test, windows-binary-wheel / wheel-py3_10-xpu-test, windows-binary-wheel / wheel-py3_12-xpu-test

Details for Dev Infra team

Raised by workflow job

atalman · 2024-11-28T21:59:36Z

@pytorchmergebot merge -f "failures are not related"

pytorchmergebot · 2024-11-28T22:01:14Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Update CUDA 12.6 to Update 3 and make cusparse-lt 0.6.3? pytorch#141365 Was going to leave some comments on pytorch#141365, but though it was just faster to open a PR here. Pull Request resolved: pytorch#141433 Approved by: https://github.com/atalman

tinglvv · 2024-12-09T10:32:45Z

.ci/docker/common/install_cuda_aarch64.sh

+function install_cusparselt_063 {
+    # cuSparseLt license: https://docs.nvidia.com/cuda/cusparselt/license.html
+    mkdir tmp_cusparselt && pushd tmp_cusparselt
+    wget -q https://developer.download.nvidia.com/compute/cusparselt/redist/libcusparse_lt/linux-x86_64/libcusparse_lt-linux-x86_64-0.6.3.2-archive.tar.xz


Missed this, should have been libcusparse_lt-linux-sbsa-0.6.3.2. This is causing error /usr/local/cuda/lib64/libcusparseLt.so: error adding symbols: file in wrong format

Skylion007 requested review from a team and jeffdaily as code owners November 23, 2024 17:00

pytorch-bot bot added the topic: not user facing topic category label Nov 23, 2024

Skylion007 requested review from atalman, eqy, jcaip, malfet, nWEIdia, ptrblck and tinglvv and removed request for jeffdaily and tinglvv November 23, 2024 17:00

pytorchbot added the open source label Nov 23, 2024

Skylion007 mentioned this pull request Nov 23, 2024

Update CUDA installation script to 12.6.3 #141365

Closed

Skylion007 force-pushed the skylion007/update-cuda-12-6-3-libraries-2024-11-23 branch 2 times, most recently from 9e38b70 to 79fbb80 Compare November 23, 2024 17:07

Skylion007 commented Nov 23, 2024

View reviewed changes

Skylion007 force-pushed the skylion007/update-cuda-12-6-3-libraries-2024-11-23 branch from 79fbb80 to 3097a63 Compare November 24, 2024 15:05

bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 26, 2024

Skylion007 force-pushed the skylion007/update-cuda-12-6-3-libraries-2024-11-23 branch from 44ab294 to 1afafa5 Compare November 26, 2024 15:59

atalman approved these changes Nov 26, 2024

View reviewed changes

tinglvv reviewed Nov 26, 2024

View reviewed changes

Skylion007 added 2 commits November 27, 2024 10:47

[BE]: Update 12.6 builds to CUDA 12.6.3

bb45024

Update nvjitlink too

b61ff06

Skylion007 force-pushed the skylion007/update-cuda-12-6-3-libraries-2024-11-23 branch from 9157523 to b61ff06 Compare November 27, 2024 15:47

tinglvv added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Nov 27, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 28, 2024

pytorchmergebot added the merging label Nov 28, 2024

pytorchmergebot removed the merging label Nov 28, 2024

pytorchmergebot added the merging label Nov 28, 2024

pytorchmergebot added the Merged label Nov 28, 2024

pytorchmergebot closed this in 7224cd4 Nov 28, 2024

pytorchmergebot removed the merging label Nov 28, 2024

tinglvv mentioned this pull request Dec 3, 2024

Add CUDA 12.6 Linux Builds to Binaries Matrix #138899

Closed

tinglvv reviewed Dec 9, 2024

View reviewed changes

atalman mentioned this pull request Dec 12, 2024

Enable CUDA 12.6 CI/CD , Disable CUDA 12.1 #138440

Open

31 tasks

tinglvv mentioned this pull request Dec 19, 2024

[aarch64] Fix libcusparselt format for CUDA sbsa docker #142363

Closed

[BE]: Update 12.6 builds to CUDA 12.6.3 #141433

[BE]: Update 12.6 builds to CUDA 12.6.3 #141433

Uh oh!

Conversation

Skylion007 commented Nov 23, 2024 • edited by tinglvv Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141433

❌ 6 New Failures, 17 Unrelated Failures

Uh oh!

Skylion007 Nov 23, 2024

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

tinglvv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tinglvv commented Nov 27, 2024

Uh oh!

Skylion007 commented Nov 27, 2024

Uh oh!

atalman commented Nov 27, 2024

Uh oh!

Skylion007 commented Nov 28, 2024

Uh oh!

Skylion007 commented Nov 28, 2024

Uh oh!

pytorchmergebot commented Nov 28, 2024

Merge started

Uh oh!

pytorchmergebot commented Nov 28, 2024

Merge failed

Uh oh!

atalman commented Nov 28, 2024

Uh oh!

pytorchmergebot commented Nov 28, 2024

Merge started

Uh oh!

tinglvv Dec 9, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Skylion007 commented Nov 23, 2024 •

edited by tinglvv

Loading

pytorch-bot bot commented Nov 23, 2024 •

edited

Loading

tinglvv left a comment •

edited

Loading